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Outline 


* Short Background on Rocks 
* Virtual Clusters 


* Practically extending a local cluster using 
Hardware” in EC2 and Condor 
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Rocks — http:// www.rocksclusters.org 


* Technology transfer of commodity clustering to application scientists 
* Rocks is a cluster/System Configuration on a CD 

— Clustering software (PBS, SGE, Ganglia, Condor, ... ) 

— Highly programmatic software configuration management 

— Put CDs in Raw Hardware, Drink Coffee, Have Cluster. 
e Extensible using “Rolls” 


* Large user community HPCE HPC: 


— Over 1PFlop of known clusters READERS’ READERS 
HOICE 
— Active user / support list of 2000+ users CHOICE AWARDS 
l AWARDS 
e Active Development 
Ni 
— 2 software releases per year 4 





— Code Development at SDSC 
— Other Developers (UCSD, Univ of Tromso, External Rolls 
e Supports Redhat Linux, Scientific Linux, Centos and Solaris 
e Can build Real, Virtual, and Hybrid Combinations (2 — 1000s) 
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EU 
TRITO 
RESOURCE 
Large Memory 
PSDAF 
e 256 GB & 512 GB 
Nodes (32 core) 


e 8TB Total 
e 128 GB/sec 









http://tritonresource.sdsc.edu 
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A Mid-Sized Cluster Resource 


Includes : Computing, Database, Storage, Virtual 
Clusters, Login, Management Appliances 










Shared Resource 
< Cluster 
=ï e 16 GB/Node 
=} * 4 - 8TB Total 
=ï e 256 GB/sec 
Dema)" ^ 20 TF 
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Large Scale Storage 
(Delivery by Mid May) 
e 2 PB ( 384 TB Today) 
e V60 GB/sec ( 7 GB/s ) 
e ^ 2600 (384 Disks Now) 























The “Downloadable” Data Center 


HA, 





ua http://tritonresource.sdsc.edu/downloads.php 


Tr Favorites | ss É Suggested Sites + Æ| Get More Add-ons + 
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= UC San Diego =. 


UCSD TRITON RESOURCE @ SDSC 


ABOUT TRITON USERS | JOBS SOFTWARE STORAGE RESOURCE STATUS 5D5C SERVICES SCIENTIFIC RESEARCH 





Triton Home K Downloads 


Triton Resource - Downloads 
Rocks Roll ISOs and Source Roll Archives On This i 


This page contains links to roll binaries and source files that will be useful for 
building a similar system to Triton Resource (or some of its components). The 
= commercial components have been separated from these downloadable ralls, 
HARDWARE DETAILS and everything here is freely available. The primary OS components are built Contact Us 
K Detailed Specs from Rocks. More details may be found on the Build 1 (BY to | oo 
Seeman aad Open a Ticket with Triton 
Visuals Roll ISOs Resource — using the 
+ Other BYO Resources The rolls listed below are available for | icket 9 
download. 





a Source File Archives 








Join the Discussion Forum 
Sign up for our Email 
Discussion List. 


Build Your Own 
Triton us BBFIP 5.3 x66 64 


Cost and Acquisition 
FAQ 


Rocks Home 


= BEAST 5.3 x86_64 

= Chemistry 5.3 x86 64 
» DataForm 5.3 x86 64 
= FFTW 5.3 x86 64 

- CCA G 9. BE GA 


A| Done, but with errors on page. F Es Internet | Protected Mode: On fa T 
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Follow Triton on Twitter 








Rocks Docs 





Key Rocks Concepts 


* Define components of clusters as Logical Appliances 
(Compute, Web, Mgmt, Login DB, PFS Metadata, PFS 
Data, ... ) 

- Share common configuration among appliances 
— Graph decomposition of the full cluster SW and Config 

Rolls are the building blocks: reusable components 

(Package + Config + Subgraph) 

* Use installer’s (Redhat Anaconda, Solaris Jumpstart) 
text format to describe an appliance configuration 
— Walk the Rocks graph to compile this definition 

* Heterogeneous Hardware (Real and Virtual HW) with 
no additional effort 
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What's in YOUR cluster? 


EE] 

d 
H 
[=] 
HI 


A = 
ï 


in 
ï 


in 


in 


in 


Ln 


M Œ D Œ m 


Ln 


in ¢ 


En 


3 
L La Co L 9 U 9 9 fo 9 Mm to tO d; U ë 
I 


4 
4 
4 
4 
4 
4 
4 
5 4 
4 
4 
4 
4 
4 
4 
4 


ttt M M D M m 





ROCKS 
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Virtual Clusters in Rocks Today 


Physical Hosting Cluster 
"Cloud Provider” 
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Require: 
1. 


Virtual Frontend 
Nodes w/disk 
Private Network 
Power 


Virtual Clusters: 

May overlap one 
another on physical HW 
Need network 
isolation 

* Maybe larger or 
smaller than physical 
hosting cluster 


How Rocks Treats Virtual Hardware 


* [t's just another piece of HW. 
— If RedHat supports it, so does 
Rocks 
* Allows mixture of real and 
virtual hardware in the same 








cluster 
— Because Rocks supports Virtual HW must meet 
heterogeneous HW clusters minimum HW Specs 
* Re-use of all of the software — 1GB memory 
configuration mechanics — 36GB Disk space* 
— E.g., a compute appliance is — Private-network Ethernet 


compute appliance, regardless 


of “Hardware” — + Public Network on 


Frontend 


* Not strict — EC2 images are 10GB 
© UC Regents 2010 





Rocks Xen Roll Supports a Broad 
Spectrum of (Virtual) Networks 













Physical Container A Physical Container B 






> Bridged 
Interfaces 
= 
y 
CO | 
L Public Network 
E 
Untagged 


Interfaces NC VLAN Tag - 2 
Isolated Private 
Networks 
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Rocks and EC2 


We can build physical hosting clusters ,multiple 


isolated virtual clusters, and hybrid mixtures of 
virtual and physical nodes: 


* Can l use Rocks to author "images" compatible 
with EC2? (We use Xen, They use Xen) 
Can l automatically integrate EC2 Virtual 


Machines into my local cluster (cluster extension) 
- Submit locally 


- My own private cloud 
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Some Challenges in EC2 


1. Defining the contents of your Virtual 
Machine (Software Stack) (Contents of AMI) 


2. Understanding limitations and execution 
model 


3. Debugging when something goes wrong 


4. Remembering to turn off your VM 
— Smallest 64-bit VM is ~$250/month running 7x24 
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Why do we even care how a 
(IAAS) Cloud image is made? 
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A: Too MANY pre-existing AMIs. No 
Systematic (Scientific) Reproducibility 


8000 










Public Amazon Instances 


E /-Sep-10 
m= 30-Dec-10 





N > a > 5 o o + N N ò> @ 6 
DS L PAF EF SF FC SP SF F L F L Z SF 48 LS $ 
S < @ P V S ë «€ ÇE FS A P OŒ O 
o oO d S AY Va o o 
g? L 2 ee N 

X SX 6 R 

LL g Ra 

x> ÇÀ 
a «S 





O UC Regents 2010 


Condor Roll 





N ò f 9 L Ê pi 3 a A j FC omputing 


* Condor 7.4.4 


* Integration with Rocks command line to do 
basic Condor configuration customization 

* To build a Condor Cluster with Rocks 
— Base, OS, Kernel, Condor Roll 
— Gives you local collector, scheduler 


* Basic, Working Configuration that can be 
customized as required. 
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_ LL 
was amazon 
p Sa The EC2 Roll 








* Take a Rocks appliance and make it compatible with EC2: 
10GB disk partition (single) 
DHCP for network 
— ssh key management 
Other small adjustments 
* Create an AMI bundle on local cluster 
- rocks create ec2 bundle 
* Upload a bundled image into EC2 
- rocks upload ec2 bundle 
e Mini-tutorial on getting started with EC2 and Rocks 
-- Register image and go. 
* Experimental: automated tunneling setup 
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All Together: Extended Condor Pool with 
Consistent System Definitions 


Job | Rocks Frontend 
Submit 


Condor 
Collector 
Scheduler 





Condor Pool with 
both local and 
cloud resources 


Cluster Private Network 
(e.g. 10.1.x.n) 


aR as E a 
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Complete Recipe 


© Upload Image to 


Rocks Frontend SE S3 
ep È ms 
annen ió Kickstart Guest VM N. 
ec2_enable=true \ 
= = 
l o 
F 


© Register Image 
as EC2 AMI 


Ë CD Bundle as S3 
| Boot AMI as an 
VAA OMANA pS Amazon Instance 





Optional: Test and 
Rebuild of Image 
Disk “Compiled” Amazon EC2 Cloud 

Storage VM Image 


y 


Local Hardware 
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At the Command Line: provided by the 
EC2 Roll/Xen Rolls 


1. rocks set host boot action=install compute-0-0 
2. rocks set host attr compute-U=-0 ec2 enable true 
3. rocks start host vm compute-O-0 

— After reboot inspect, then shut down 
4. rocks create ec2 bundle compute-0-0 
5. rocks upload ec2 bundle compute-0-0 <s3bucket> 
6. ec2-register <s3bucket>/image.manifest.xml 
7. ec2-run instances <ami> 


O UC Regents 2010 a 


Summary 


Easily Extend your Condor pool into EC2 

— Others can do this as well 

- Condor supports the public/private network duality of 
EC2 

* Have your software on both local cluster and 
remote VM in EC2 

* Mix and match 
— Local Physical, Local Virtual, Remote Virtual 

* Familiar tools and paradigms for cloud-hosted 

VMs. 
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